fix: make E2E tests more stable #124
Conversation
This attempts to prevent tests from failing because of timeouts
Highlighted by Playwright accessibility checks
|
I'm testing these right now, but some are failing for me. Currently working on a fix. |
There was a problem hiding this comment.
One test failed for me: “a user can log out and log in again” on Mobile Chrome. The failing step is logOut at fixtures.ts:86.
I ran the same test 10 times and got 6 passes and 4 fails. AI suggested the flaky test is a timing issue, which fits the failure snapshot, where the menu has already closed when the test looks for “Log out”.
|
Try now @hestro |
|
@johahor However, when I ran all the e2e tests three times, I got three different results: Run 1: Run 2: Run 3: There still seem to be some flaky tests... |
|
We can reduce the amount of workers even more, because the timeouts/failures are usually caused by that. @hestro could you try decrasing the number 4 to 1 and see if that fixes it? |
|
@martakam |
How many workers are able to run without timeouts depends on the system, so the safe approach is to use only one. This should be documented in a separate commit
|
The worker count is down to 1. The choice should be documented in #134. |
Many E2E tests failed because of timeouts, likely caused by running 8 workers running browsers at the same time.
This has now been decreased to 4.
Some tests also failed because we have a different menu in the header on mobiles now. This has been fixed by opening the menu and checking its content if we're testing on mobile screen sizes.
Furthermore, some smaller fixes highlighted by the accessibility checks from Axe have been implemented, and a slight refactoring of the E2E tests has been done.
All of these pass for me locally now, even though one must wait a little longer for the results. Please let me know if everything looks good for you guys as well.
Closes #120